"""
隐藏链接反爬虫
爬取目标站点商城列表中每个商品详情页，拿到详情页的响应正文
http://www.porters.vip:8202/
"""
import time
import requests
from lxml import etree
from urllib.parse import urljoin


url = 'http://www.porters.vip:8202/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/92.0.4515.159 Safari/537.36 '
}
res = requests.get(url, headers=headers)
parser = etree.HTMLParser(encoding='utf-8')
html = etree.HTML(res.text, parser=parser)
goods = html.xpath('//*[@class="col-md-3"]')
for item in goods:
    link = item.xpath('.//*[contains(@class, "caption")]//p//a/@href')[0]
    detail_url = urljoin(url, link)
    detail = requests.get(detail_url, headers=headers).text
    print(detail)
    time.sleep(0.5)